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translational  and  rotational  velocities  that  in  combination  with  the  Brightness- 
Change  Constraint  Equation  solves  the  general  motion  vision  problem,  arbitrary 
motion  with  respect  to  an  arbitrary  rigid  environment. 

Avoiding  correspondence  and  optical  flow  has  been  the  motivation  behind  the 
direct  methods  because  both  solving  the  correspondence  problem,  and  computing 
the  optical  flow  reliably,  have  proven  to  be  rather  difficult  and  computationally 
expensive. 

Recently  direct  motion  vision  methods,  which  directly  use  the  image  brightness 
information  such  as  temporal  and  spatial  brightness  gradients  directly,  have  used 
the  Brightness-Change  Constraint  Equation  for  solving  the  motion  vision  problem 
in  special  cases  such  as  Known  Depth ,  Pure  Translation  or  Known  Rotation ,  pure 
Rotation ,  Planar  World  and  Quadratic  Patches.  In  contrast  to  these  solutions, 
our  fixation  method  does  not  put  such  severe  restrictions  on  the  motion  or  the 
environment. 
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1  INTRODUCTION 

In  motion  vision,  the  goal  is  to  recover,  from  time  varying  images,  the 
relative  motion  between  a  viewer  and  an  environment  as  well  as  the  structure 
of  objects  in  the  environment.  A  survey  of  previous  literatures  on  machine 
vision  is  given  by  Barron  [10].  Some  of  the  current  issues  in  image  flow  theory 
and  motion  vision  are  discussed  by  Waxman  &  Wohn  [32]  and  Aloimonos  & 
Shulman  [4].  Much  of  the  earlier  work  on  recovering  motion  has  been  based 
on  either  establishing  correspondences  between  the  images  of  prominent  fea¬ 
tures  (points,  lines,  contours,  and  so  on)  in  an  image  sequence  (for  example. 
Prazdny  [27],  Uilman  [30,  31j,  Longuet-Higgins  [19],  and  Aloimonos  &  Basu 
[2])  or  establishing  the  velocity  of  points  over  the  whole  image,  commonly 
referred  to  as  the  optical  flow  (for  example,  Ballard  Kimball  [5],  Bruss  & 
Horn  [11],  and  Adiv  [1]). 

In  general,  identifying  features  here  means  determining  gray-level  corners. 
For  images  of  smooth  objects,  it  is  difficult  to  find  good  features  or  corners. 
Further,  the  correspondence  problem  has  to  be  solved,  that  is,  feature  points 
from  consecutive  frames  have  to  be  matched. 

The  computation  of  the  local  flow  field  exploits  a  constraint  equation 
between  the  local  brightness  changes  and  the  two  components  of  the  optical 
flow.  This  only  gives  the  components  of  flow  in  the  direction  of  the  brightness 
gradient.  To  compute  the  full  flow  field,  one  needs  additional  constraints  such 
as  the  heuristic  assumption  that  the  flow  field  is  locally  smooth  (Hildreth  [14], 
and  Horn  &  Schunck  [15]).  This  leads  to  an  estimated  optical  flow  field  that 
is  not  the  same  as  the  true  motion  field. 

Both  solving  the  correspondence  problem,  and  computing  optical  flow 
reliably,  have  proven  to  be  rather  difficult  and  computationally  expensive. 
This  has  motivated  the  investigation  of  direct  methods  which  use  the  image 
brightness  information  directly  to  recover  motion. 

Recently  direct  motion  vision  methods  have  used  the  Brightness-Change 
Constraint  Equation  (BCCE)  for  solving  the  motion  vision  problem  in  special 
cases  such  as  Known  Depth  [15],  Pure  Translation  or  Known  Rotation  [17, 
24,  16],  Pure  Rotation  [17],  Planar  World  [23]  and  Quadratic  Patches  [22].  In 
this  work,  a  method  called  fixation  has  been  introduced  which  in  combination 
with  the  brightness-change  constraint  equation  solves  the  direct  motion  vision 
problem  of  arbitrary  motion  with  respect  to  an  arbitrary  rigid  environment. 
That  is,  it  recovers  the  shape,  rotational  velocity  and  translational  velocity 


2 


in  the  general  case.  In  contrast  to  the  tracking  methods  presented  in  [3,  9, 
28,  29],  our  fixation  method  is  not  only  different  but  also  is  general.  For 
example,  Aloimonos  &  Tsakiris  [3]  propose  a  method  for  tracking  a  target 
of  known  shape.  Badopadhay,  Chandra  &  Ballard  [9]  use  optical  flow  for 
tracking.  Sadini  &  Tistarelli  [29]  do  tracking  for  the  special  case  in  which 
the  component  of  rotational  velocity  along  the  optical  axis  is  zero. 

A  block  diagram  of  the  ideas  behind  this  work  is  shown  in  figure  1.  We 
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Figure  1:  A  block  diagram  of  the  fixation  method  modules. 

start  with  a  brief  review  of  the  BCCE  in  section  2.  Then  in  section  3,  we 
show  that  by  choosing  an  interest  point  in  the  environment.  R0.  and  knowing 
the  component  of  rotational  velocity  along  the  position  vector  associated 
with  that  interest  point,  ■jj we  can  obtain  a  Fixation  Constraint  Equation 
(FCE)  between  the  rotational  velocity  u  and  the  translational  velocity  t  just 
by  keeping  the  image  of  the  interest  point  stationary  in  the  image  plane. 
Section  4  shows  how  the  fixation  constraint  equation  can  be  combined  with 
the  BCCE  and  applied  to  fixated  images  in  order  to  find  t.  u  and  depth  Z  in 
the  general  case.  Recovering  u,ro.  needed  in  the  fixation  constraint  equation, 
and  finding  the  components  of  fixation  velocity.  u0  and  r0.  necessary  for 
obtaining  a  fixated  2nd  image,  are  discussed  in  section  5.  In  order  to  apply 
the  fixation  constraint  equation,  a  sequence  of  two  fixated  images  is  needed. 
Initial  1st  image  can  be  used  directly  and  section  6  shows  how  a  fixated  2nd 
image  is  obtained  from  the  initial  2nd  image. 
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2  THE  BRIGHTNESS  CHANGE  CONSTRAINT  EQUATION 


Using  a  viewer-based  coordinate  system  which  is  adopted  from  Longuet- 
Higgins  and  Prazdny  [18]  is  very  common  in  direct  motion  vision.  Figure  2 
depicts  the  coordinate  system  under  consideration. 


In  this  coordinate  system,  a  world  point 

R  =  (XY  Z)T 

(1) 

is  imaged  at 

r  =  (x  y  1)T. 

(2) 

Figure  2:  The  viewer-centered  coordinate  system.  The  translational  velocity 
of  the  camera  is  t  =  (U  V  IF)7,  and  its  rotational  velocity  is  u  =  (A  B  C)T . 

That  is,  the  image  plane  has  the  equation  Z  =  1  or  in  other  words  the  focal 
length  is  1.  The  origin  is  at  the  projection  center  and  the  Z-axis  runs  along 
the  optical  axis.  The  X-  and  Y-  axes  are  parallel  to  the  x-  and  y-  axes  of 
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the  image  plane.  Image  coordinates  are  measured  relative  to  the  principal 
point ,  the  point  (0  0  1)T  where  the  optical  axis  pierces  the  image  plane.  The 
position  vectors  r  and  R  are  related  by  the  perspective  projection  equation 


where  R-  z  =  Z  and  z  denotes  the  unit  vector  in  the  Z  direction. 

When  the  observer  moves  with  instantaneous  translational  velocity  t  = 
( U  V  W)T  and  instantaneous  'otational  velocity  to  =  (A  B  C)T  relative  to 
a  rigid  environment,  then  the  time  derivative  of  the  vector  R  can  be  written 
as 

Rt  =  —  t  —  uj  x  R.  (4) 

The  motion  of  the  world  point  R  results  in  motion  of  the  corresponding 
image  point  r.  It  can  be  shown  [23,  21]  that  the  motion  field  in  the  image 
plane  is  obtained  by  differentiating  eqn.  (3)  with  respect  to  time  as 

d  (  R  \  z  x  (Rf  x  r) 

f£  =  Jt  \R~~i J  =  rTI 


Substituting  for  R,  r  and  R<  from  equations  (1),  (2)  and  (4)  into  eqn.  (5) 
gives,  [18,  11] 


rt 


+  Axy  _  B{x2  +  1)  +  Cy 
=AL±uK  _  Bxy  +  ,4(y2  +  l)  —  Cx 


(6) 


This  result  is  just  the  parallax  equations  of  photogrammetry  that  occur  in 
the  incremental  adjustment  of  relative  orientation  [13,  20].  It  shows  how, 
given  the  environment  motion,  the  motion  field  can  be  calculated  for  every 
image  point. 

Image  brightness  changes  are  primarily  due  to  the  relative  motion  between 
an  environment  and  an  observer  provided  that  the  surfaces  of  the  objects  have 
sufficient  texture  and  the  lighting  condition  varies  slowly  enough  both  spa¬ 
tially  and  with  time.  In  this  case,  brightness  changes  due  to  changing  surface 
orientation  and  changing  illumination  can  be  neglected.  Consequently,  we 
may  assume  that  the  brightness  of  a  small  patch  on  a  surface  in  the  scene 


5 


does  not  change  during  motion.  Then  expansion  of  the  total  derivative  of 
brightness  E  leads  to 

dE 

—  =  Et  +  xtEx  +  ytEy  =  01  (7) 

which  is  referred  to  as  the  Brightness  Change  Constraint  Equation  (BCCE) 
[15].  By  substituting  for  xt  and  yt  from  eqn.  (6)  into  eqn.  (7),  we  obtain  the 
brightness  change  constraint  equation  for  rigid  body  motion  [23],  namely 

Et  +  v  ■  u  4 — =  0  (8) 

where  the  auxiliary  vectors  s  and  v  are  defined  as 


xEx  +  yEy 


/  +Ey  +  y(xEx  +  yEy)  \ 

v=  -Ex  -  x(xEx  +  yEy)  .  (10) 

\  yEx  -  xEy 

Considering  that  s  •  r  =  0,  v  •  r  =  0  and  s  •  v  =  0,  these  three  vectors  thus 
form  an  orthogonal  triad.  The  vectors  s  and  v  represent  inherent  properties 
of  the  image.  Also  it  can  be  shown  that  v  =  r  x  s.  The  vector  s  indicates 
the  directions  in  which  translation  of  a  given  magnitude  will  contribute  max¬ 
imally  to  the  temporal  brightness  change  of  a  given  picture  cell.  The  vector 
v  plays  a  similar  role  for  rotation. 

The  brightness  change  constraint  equation  is  unchanged  if  we  scale  both 
Z  and  t  by  the  same  scale  factor.  We  conclude  that  we  can  determine  only 
the  direction  of  translational  velocity  and  the  relative  depth  of  points  in  the 
scene.  This  well-known  ambiguity  is  referred  to  as  the  scale-factor  ambiguity 
in  motion  vision. 

'To  account  for  smooth  variations  in  the  image  brightness  due  to  other  factors  such  as 
shading,  spatial  and  temporal  illumination  changes,  and  variations  in  reflectance  proper¬ 
ties,  the  BCCE  can  be  extended  to 

Et  +  x,Ez  +  ytEy  =  mtE  +  c, 

where  in  general  mt  and  c(  are  time  and  position  dependent  [12,  25,  26].  For  simplicity, 
we  have  not  used  this  extension  here  but  we  may  use  it  for  implementation. 
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3  FIXATION  FORMULATION 

Our  common  visual  experience  suggests  that  fixation  may  play  an  impor¬ 
tant  role  in  the  analysis  of  moving  objects.  When  we  want  to  understand  the 
motion  of  an  object  we  do  not  keep  our  eyes  and  head  stationary  in  front  of 
the  moving  object.  Instead  our  head  and/or  eyes  follow  the  moving  object, 
in  order  to  keep  the  image  of  a  point  of  interest  stationary  in  the  retina. 
There  are  also  some  formal  studies  that  support  such  observations  [6,  7,  8]. 
Consequently  in  this  computer  vision  work,  the  fixation  is  defined  as: 

Given  two  subsequent  images,  initial  1st  and  2nd  images ,  and  a 
point  in  the  1st  image,  find  a  sequence  of  two  fixated  images  by 
obtaining  a  new  image,  fixated  2nd  image ,  such  that  the  image 
of  the  selected  point  in  the  new  image  is  located  at  its  original 
position  as  in  the  initial  1st  image. 

As  shown  in  figure  3,  we  refer  to  this  selected  image  point  as  the  fixation 
point ,  r0,  and  to  its  corresponding  point  on  the  object  as  the  interest  point , 

Ro- 


3.1  Derivation  of  General  Fixation  Constraint  Equation 

In  a  sequence  of  two  fixated  images,  at  fixation  point  we  should  have 

r,j  =  0  (11) 

where  rot  is  the  time  derivative  of  the  fixation  point  vector  and  similar  to 
eqn.  (5)  can  be  written  as 


z  x  (Ro!  x  r0) 

r°'  =  — R,  —  •  (12) 

Rot  is  the  time  derivative  of  the  interest  point  vector.  Combination  of  equa¬ 
tions  (11)  and  (12)  shows  that  for  fixation  we  need  to  have 

z  x  (R0(  x  r0)  =  0.  (13) 


In  other  words,  we  want  to  find  out  when  R0(  x  r0  is  zero  or  parallel  to  z. 
For  Rot  x  r0  to  be  parallel  to  z,  we  should  have  r0  perpendicular  to  z  which 
is  impossible  with  a  finite  field  of  view  (FOV).  so  only  R0t  x  r0  =  0  applies. 
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Figure  3:  In  fixation  the  fixation  point ,  the  image  of  the  interest  point,  is 
kept  stationary  in  the  image  plane  despite  the  relative  motion  between  the 
camera  and  the  environment. 

Conclusively,  considering  that  R0  and  r0  have  the  same  direction,  eqn.  (13) 
is  simplified  as 

R0j  x  R0  =  0  (14) 

Now  substituting  for  Rot  =  —  t  —  u  x  R0,  eqn.  (4),  into  (14)  gives 

(u  xR0)  x  R„  +  t  x  R,  =  0.  (15) 

Expansion  of  eqn.  (15)  by  using  (a  x  b)  x  c  =  (c  •  a)b  —  (c  •  b)a  results  in 
(R0  •  u;)R0  —  (R0  •  Ro)a>  +  t  x  R0  =  0.  (16) 

As  long  as  the  translational  velocity  t  is  neither  zero  nor  parallel  to  the 
interest  point  vector  R0.  then  any  vector,  including  u,  can  be  expressed  in 
terms  of  the  triad  of  vectors  R0.  t  x  R0  and  t.  So  we  can  write  u>  in  its 
general  form  as 


u>  =  qR0  +  d(t  x  R0)  +  7t 


(17) 
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where  a,  ft  and  7  are  constants  to  be  determined.  Later  in  this  section  we 
will  consider  the  special  cases  where  t  is  zero  or  parallel  to  R0  by  defining  u> 
based  on  another  triad. 

Substituting  for  u  from  eqn.  (17)  into  eqn.  (16)  gives 

[1  -  ft (H0  •  Ro)](t  X  R0)  +  7(Ro  •  t)Ro  -  7(Ro  ■  Ro)t  =  0.  (18) 


Now,  we  should  find  the  constants  ft  and  7  such  that  eqn.  (18)  holds  without 
putting  any  restrictions  on  R0  and  t.  We  start  by  finding  the  dot  product  of 
eqn.  (18)  by  t  x  R0  that  results  in 


[1  —  /3(R0  •  Ro)]|jt  x  R0||2  =  0. 


Equation  (19)  will  hold  without  restricting  R0  and  t  if 


ft  = 


1 

IIRoll2' 


(19) 


(20) 


Another  possibility  for  satisfying  eqn.  (19)  is  to  have  ||t  x  R0||  =  0  which 
implies  that  t  =  0,  Rc  =  0  or  t  is  parallel  to  Rc.  But  R0  cannot  be  zero  and 
also  we  assumed  that  here  t  is  neither  zero  nor  parallel  to  Ro.  As  a  result, 
||t  x  R0  ||  cannot  be  zero.  Similarly  the  dot  product  of  eqn.  (18)  by  t  gives 


t(Ro  •  t)(R0  •  t)  -  7(Ro  •  Ro)(t  •  t)  =  0.  (21) 


Knowing  that  (a  x  b)  •  (c  x  d)  =  (c  •  a)(b  •  d)  -  (d  •  a)(b  •  c).  eqn.  (21)  can 
be  simplified  as 

7||txRo||2  =  0.  (22) 

We  discussed  that  ||t  x  R0||  cannot  be  zero  here,  so  eqn.  (22)  is  satisfied  only 
if  7  is  zero 

7  =  0.  (23) 


Substituting  for  3  from  eqn.  (20)  and  7  from  eqn.  (23)  into  eqn.  (17) 
gives 


u  =  qRo  4- 


1 


(t  x  R,) 


(24) 


where  a  is  still  unknown.  This  means  that  the  component  of  the  rotational 
velocity  along  R0  cannot  be  determined  by  the  fixation  formulation.  Physi¬ 
cally  this  makes  sense  because  the  rotational  velocity  along  R0.  denoted  by 
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wro,  does  not  move  the  fixation  point.  This  observation  leads  us  to  find 
in  a  separate  step  before  using  the  fixation  formulation  results.  Derivation 
of  will  be  shown  in  section  5.  Finally  the  general  fixation  constraint 
equation  (GFCE)  is  written  as 

w  =  wRoR0  +  x  R0)  (25) 

where  t  is  the  translational  velocity  and  R0  =  f0  is  the  unit  vector  along  the 
position  vector  of  an  arbitrary  fixation  point,  a  point  in  the  image  chosen  for 
fixation. 

3.2  Derivation  of  Special  Fixation  Constraint  Equation 

When  the  translational  velocity  t  is  zero  or  parallel  to  the  interest  point 
vector  R0,  eqn.  (16)  is  simplified  as 

(Ro-u)Ro-(Ro-R>V  =  0.  (26) 

We  define  u  based  on  the  triad  consisting  of  vectors  R0.  x.  and  x  x  R0  as 

a;  =  /R0  +  m(x  x  R0)  +  nx  (27) 

where  /,  m,  and  n  are  constants  to  be  determined.  He-e  we  assume  that  R0 
is  not  parallel  to  x.  This  is  a  reasonable  assumption  because  otherwise  we 
should  at  least  have  a  field  of  view  of  ISO0  to  be  able  to  choose  an  awkward 
point  of  interest  along  the  x-axis,  which  results  in  a  fixation  point  at  infinite 
distance  from  the  principal  point  and  near  the  border  of  an  infinite  image 
plane. 

Substituting  for  u>  from  eqn.  (27)  into  eqn.  (26)  gives 

[mR0  •  (x  x  R0)  +  n(Rc  •x)]R0  —  m(R0  •  R0Hx  x  R0)  —  n.(Rc  •  R«)x  =  0.  (28) 

The  dot  product  of  eqn.  (2S)  with  (x  x  R0)  results  in 

—  m(Rc  •  R0)||x  x  R0||2  =  0.  (29) 

Considering  that  R,  cannot  be  either  zero  or  parallel  to  x.  eqn.  (29)  is 
satisfied  only  if  m  is  zero 


m  =  0. 


(30) 
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Substituting  for  m  into  eqn.  (28)  and  finding  its  dot  product  by  x  results  in 

n(R0  •  x)(Ro  •  x)  -  n(R0  •  Rc)(x  •  x)  =  0.  (31) 

Using  (a  x  b)  •  (c  x  d)  =  (c  •  a)(b  •  d)  —  (d  •  a)(b  •  c),  eqn.  (31)  can  be  written 
as 

n||x  x  R0||2  =  0.  (32) 

Again  R0  cannot  be  either  zero  or  parallel  to  x.  As  a  result,  eqn.  (32)  will 
hold  for  arbitrary  R0  if  n  =  0.  Substituting  for  n  and  m  into  eqn.  (27)  gives 

v  =  IRo  (33) 

where  l  is  still  unknown.  We  can  substitute  wroR0  for  /R0.  It  will  be  shown 
later  how  u>ro  is  found  separately.  As  a  result  for  the  special  cases  we  obtain 
the  special  fixation  constraint  equation  (SFCE)  as 

^  =  ^R0Ro  (34) 

which  means  that  when  the  translational  velocity  t  is  zero  or  parallel  to  R0 
then  the  corresponding  rotational  velocity  may  only  have  a  component  along 

R0. 

Our  method  for  deriving  the  SFCE,  eqn.  (34),  is  not  different  from  what 
we  did  for  deriving  the  GFCE,  eqn.  (25).  In  fact,  eqn.  (34)  is  a  special  case 
of  eqn.  (25).  But  we  could  not  directly  derive  eqn.  (34)  from  eqn.  (25) 
because  eqn.  (25)  was  derived  based  on  the  assumption  that  t  is  neither 
zero  nor  parallel  to  R0.  As  a  result,  for  implementation  it  is  enough  to  use 
the  GFCE,  eqn.  (25),  without  knowing  whether  the  present  condition  is  the 
special  case  or  not. 

4  SOLVING  THE  GENERAL  DIRECT  MOTION  VISION  PROB¬ 
LEM 

Here  we  assume  that  we  are  given  a  sequence  of  fixated  images.  In  other 
words  we  have  made  the  fixation  point  stationary  in  our  image  sequence. 
This  can  be  done  by  finding  the  fixation  velocity ,  the  apparant  velocity  at 
the  fixation  point  in  the  1st  image,  as  given  in  section  5.  Then  the  shifting 
method  explained  in  section  6  can  be  used  for  generating  a  new  image,  fixated 
2nd  image ,  in  which  the  fixation  point  is  located  at  the  same  position  as  in 
the  initial  1st  image. 
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We  start  by  studying  the  general  case  where  the  translational  velocity 
t  is  neither  zero  nor  parallel  to  the  interest  point  vector  R0.  Then  we  will 
consider  the  special  cases  of  t  separately. 

Substituting  for  u;  from  the  general  fixation  constraint  equation  (25)  into 
the  brightness-change  constraint  equation  (8)  gives 

Et  +  wr,v  •  Ro  +  pj  (v  •  (t  x  R0)J  +  ~(s  •  t)  =  0.  (35) 

Knowing  that  a  •  (b  x  c)  =  (a  x  b)  •  c  and  doing  some  manipulations  on  eqn. 
(35)  results  in 

K  +  [;|s  -  p-jj-(v  x  ^°)1  • t  =  0  (36) 

where  E't  is  a  notation  for  £* +wr.0v-R0  which  can  be  computed  at  any  pixel. 
In  general,  eqn.  (36)  can  be  solved  numerically  for  t  and  Z  using  images  of 
any  size  and  with  any  field  of  view  (FOV).  In  the  following,  a  closed  form 
solution  is  presented  for  the  case  that  a  small  patch  around  the  fixation  point 
is  used  or  the  field  of  view  is  small  and  the  whole  image  is  used. 

We  know  that  at  the  fixation  point  v  x  R0  =  vD  x  R0  =  0.  For  a  small 
field  of  view,  the  product  of  v  x  R0  will  be  negligible.  Even  for  an  image 
with  a  large  field  of  view  this  is  still  true  for  the  image  area  near  the  fixation 
point.  As  a  result,  for  these  cases,  eqn.  (36)  can  be  written  as 

E't  +  -j(s  •  t)  as  0  (37) 

which  con  be  solved  similar  to  the  pure  translation  case  [17,  24.  16].  Questions 
such  as  “How  large  can  the  FOV  or  fixation  patch  be  to  guarantee  the  validity 
of  this  approximation?"  can  be  answered  by  implementation  results. 

Considering  that  the  depth  range  is  finite,  we  can  solve  eqn.  (37)  by 
minimizing 

J I  Z2 did ij  =  J J ( )2dxdy  =  J  (38) 

with  respect  to  t.  In  other  words,  we  are  looking  for  the  true  motion  t  which 
minimizes  the  sum  of  squares  of  the  depth  estimates  over  the  image  of  a 
scene  with  a  finite  depth  range.  In  order  to  avoid  the  trivial  solution  t  =  0. 
a  constraint  such  as  ||t||  =  1  is  put  on  this  minimization  problem.  This  is  a 
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valid  constraint  on  t  because  due  to  the  scale  factor  ambiguity  we  can  only 
find  the  direction  of  t.  This  constraint  on  t  can  be  written  as 

tTt  =  1.  (39) 

Moreover  we  can  rewrite  J  as 

J  =  tT  Mt  (40) 

where  M  is  a  fully  computable  3x3  matrix 

M  =  JJ(±)hsTdXdy.  (41) 

Minimizing  J  in  eqn.  (40)  under  the  constraint  eqn.  (39)  is  an  ordinary  cal¬ 
culus  constrained  minimization  problem  which  can  be  solved  by  minimizing 

/( t,  A)  =  tT Mt  +  A(1  -  trt)  (42) 


with  respect  to  t  and  the  Lagrange  multiplier  A.  Then  we  will  have 


m 

dt 


=  2 Mt  -  2At  =  0 


(43) 


which  is  simplified  as 

Mt  =  At.  (44) 

Equation  (44)  is  an  eigenvalue  problem  where  A  is  an  eigenvalue  of  the  known 
matrix  M  and  t  is  the  corresponding  eigenvector.  Substituting  Mt  from 
eqn.  (44)  into  eqn.  (42)  gives  /  =  A  which  implies  that  under  the  given 
constraint  tT Mt  is  minimized  when  the  smallest  of  three  eigenvalues  is  used 
for  calculating  the  eigenvector  t. 

It  is  concluded  that  the  fixation  method  can  be  used  for  solving  the  mo¬ 
tion  vision  problem  in  its  general  case.  The  translational  velocity  t  can  be 
calculated  from  eqn.  (44)  by  using  the  smallest  eigenvalue.  Then  we  can  use 
eqn.  (36)  for  finding  the  environment  depth 


(45) 
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and  finally  eqn.  (25)  gives  the  rotational  velocity  u 

W  =  WIll^  +  pj(tx^)'  (46) 

The  total  rotational  velocity  of  the  vehicle  with  respect  to  the  environ¬ 
ment  is  obtained  by  adding  u)  to  the  equivalent  rotational  velocity  f l  given  in 
section  6.  It  can  be  seen  that  for  the  general  case,  the  fixation  formulation 
lets  us  find  the  shape  and  motion  parameters  based  on  an  arbitrary  choice 
of  interest  point  R0. 


4.1  Special  Cases:  t  Is  Zero  or  Parallel  to  R<> 

When  the  translational  velocity  t  is  zero,  we  showed  that  the  rotational 
velocity  u  has  only  a  component  along  R0.  In  this  case  we  basically  cannot 
obtain  any  estimation  for  the  depth  Z  but  there  are  methods  for  finding  the 
rotational  velocity  u>  [17].  For  the  other  special  case  where  t  is  parallel  to 
Ro,  we  substitute  for  u  from  eqn.  (34)  into  the  BCCE  eqn.  (8)  to  obtain 


e: 


+  j(s-t)  =  0 


(47) 


where  E[  is  a  notation  for  the  computable  value  of  Et  +  u;Rov  •  R0.  Because 
no  approximation  is  involved  in  deriving  eqn.  (47),  an  exact  closed  form 
solution  exists  for  t  and  Z  without  a>.y  restriction  on  the  field  of  view  or 
the  image  size.  This  exact  solution  for  finding  t  and  Z  is  the  same  as  the 
solution  given  in  the  general  case,  starting  from  eqn.  (38). 


5  COMPUTING  THE  FIXATION  VELOCITY  AND  u.Ro 

The  fixation  formulation  is  based  on  the  assumption  that  the  fixation 
point  remains  stationary  in  a  sequence  of  fixated  images.  We  use  the  term 
fixation  velocity  to  refer  to  the  apparent  velocity  at  the  fixation  point  in 
the  initial  1st  image.  We  also  represent  x  and  y  components  of  the  fixation 
velocity  by  u0  and  v0  respectively.  The  basic  fixation  requirement,  a  sequence 
of  two  fixated  images  in  which  r0(  =  0.  can  be  satisfied  by  finding  u0  and 
v0,  and  then  using  these  components  for  obtaining  a  new  image,  fixated  2nd 
image.  The  shifting  method  for  obtaining  the  fixated  2nd  image  is  explained 
in  the  next  section. 
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We  also  saw  that  the  component  of  the  rotational  velocity  along  R0,  u;r0, 
cannot  be  calculated  from  the  fixation  formulation  because  this  component 
does  not  move  the  fixation  point.  Here,  we  will  introduce  algorithms  which 
can  be  used  for  finding  both  u^r0  and  the  components  of  the  fixation  velocity. 

If  we  assume  that  depth  is  approximately  constant  on  a  small  patch 
around  the  fixation  point,  the  fixation  patch,  then  u0  and  v0  will  be  ap¬ 
proximately  constant  on  this  patch.  Possible  sensitivity  of  this  assumption 
to  special  cases  such  as  slanted  surfaces  can  be  checked  by  implementation. 
Moreover  the  motion  field  velocity  due  to  the  component  of  the  rotational 
velocity  of  the  camera  with  respect  to  the  environment  along  R0  is  given 
by  -(«R0  x  r)  =  -u>r0(R0  x  r)  =  ~^{r0  x  r)  because  R0  =  f0  is  the 
unit  vector  along  r0.  Knowing  that  r0  =  (x0  y0  l)r  and  r  =  (x  y  1)T,  the 
components  of  the  total  motion  field  velocity  along  the  x  and  y  axes,  due  to 
fixation  velocity  and  cjr0,  are  given  by 


JRo 


Xt  -  U°  ijrTit x 

JR„ 


(re  X  r)  =  Uo  -  uR„(yo  -  y) 


Vt 


=  V0  - 


|r0| 


y  •  (r0  x  r)  =  v0  -  J:-R0(x  -  x0) 


(48) 


where  x  and  y  are  the  unit  vectors  along  the  x  and  y  axes  and  u>r0  is  a 
notation  for  p^jj-.  Substituting  for  xt  and  yt  from  the  above  equations  into 
the  BCCE,  eqn.  (7),  gives 


U0  -  WR0(y0  -  y)\Ex  +  [v0  -  ^R0(x  -  x0)\Ey  +  Et  =  0.  (49) 


Due  to  noise,  eqn.  (49)  does  not  necessarily  hold  for  any  r  so  we  try  to  find 
uQ,  v0  and  u>ro  by  minimizing  the  sum  of  squares  of  errors  over  the  fixation 
patch,  denoted  by  p.  In  other  words  we  want  to  minimize 


Uo  -  ^nfiyo  ~  y))Ex  +  (u0  -  u^’Ro(x  -  xa))Ey  +  Et)2dxdy 


(50) 


with  respect  to  u0,  v0  and  wro  which  results  in  a  system  of  three  linear 
equations  which  can  be  solved  for  the  three  unknowns 


an 

an 

Ul3 

i  U0 

a2l 

a  2  2 

U23 

vo 

<*31 

U32 

U33 

\  -'’It, 

Matrix  A  is  symmetric  and  its  elements  are  given  by 


(51) 
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a12 

=  a21  -- 

ffp  ExEydx  dy 

<*13 

=  031  = 

-ffp  Ex[Ex(y0  -y)  +  Ey{x 

—  x0)}dx  dy 

<*23 

=  <*32  = 

-ffp  Ey[Ex(y0  -y)  +  Ey{x 

—  x0)\dx  dy 

<*11  = 

ffp  Exdx  dy 

a22  = 

ffp  Eydx  dy 

> 

<*33  = 

ffP[Ex(y0  -y)  +  Ey(x  -  x 

0)]2dz  dy 

and  the  elements  of  vector  C  are  as  follows: 


Ci  =  —ffp  EtExdx  dy 

<  c2  =  -ffpEtEydxdy  (53) 

.  c3  =  ffp  Et[Ex{y0  -y)  +  Ey{x  -  x0)]dx  dy. 

Considering  that  the  fixation  point  coordinates  x0  and  y0  are  known,  then 
the  sets  of  equations  (52)  and  (53)  show  that  the  elements  of  matrix  A  and 
vector  C  can  be  calculated  easily. 

In  the  special  case  where  the  fixation  point  is  at  the  principal  point, 
£0  =  J/o  =  0,  the  a,j  are  simplified  as: 


<*12 

=  <*21 

<*13 

=  <*31 

<*23 

=  <*32 

<*11 

<*22 

<*33 

and  c,  are  given  as  follows: 


=  ffp  ExEydx  dy 
=  ffp  Ex ( y Ex  -  xEy)dx  dy 
=  ffp  Ey{yEx  -  xEy)dx  dy 
=  ffp  Ef.dx  dy 
=  ffpEydxdy 
=  ffp{yEx  -  xEy)2dx  dy 


(54) 


f  ci  =  —ffp  EtExdx  dy 

S  c2  =  -ffpEtEydxdy  (55) 

1  c3  =  -ffp  Et(yEx  -  xEy)dx  dy. 

After  finding  <Dr.0,  we  can  easily  calculate  as 

WR,  =  ^R.0\/x°  +y  0  +  *•  (56) 

When  the  fixation  point  is  at  the  principal  point,  u;ro  is  exactly  the  same  as 

^R„- 
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6  OBTAINING  A  FIXATED  2ND  IMAGE 

The  fixation  method  assumes  that  a  sequence  of  two  images  are  available 
in  which  the  fixation  point  is  kept  stationary.  Referring  to  figure  1,  we  are 
given  two  original  images.  The  1st  original  image  is  used  directly  but  we 
need  to  find  a  fixated  2nd  image. 

Physical  rotation  of  the  camera  with  respect  to  the  vehicle  is  a  hardware 
solution  to  this  problem  which  is  basically  a  tracking  problem.  Considering 
that  in  general  the  interest  point  has  a  relative  motion  with  respect  to  the 
vehicle,  the  fixated  2nd  image  cannot  be  obtained  in  one  step.  As  a  result,  a 
feedback  loop  is  required  for  the  camera  rotation  system  to  compensate  for 
the  errors  resulted  from  the  new  position  of  the  fixation  point.  This  hardware 
approach  is  avoided  not  only  because  of  the  errors  involved  but  also  because 
of  the  concern  about  the  real  time  applications. 

In  the  following  we  will  show  how  a  fixated  2nd  image  can  be  obtained  by 
applying  a  compensating  rotation  to  the  initial  2nd  image  through  software. 
It  is  assumed  that  the  fixation  velocity  has  been  already  computed,  eqn.  (51). 
We  introduce  an  equivalent  rotational  velocity  fi  =  (f2x,  f2v,  f2z)  which  could 
result  in  the  same  fixation  velocity  ( u0,v0 )  at  the  fixation  point  (x0,y 0). 
According  to  eqn.  (6),  the  components  of  Q,  must  satisfy  the  following  set  of 
equations: 

f  u0  —  x0y0Qx  (x0  T  l)f!y  T  J/ofl;  /--.v 

\  v0  =  (y*  +  l)f2r  -  x0y0Uy  -  x0Q:.  0< 

Among  infinite  number  of  12  that  satisfy  the  system  of  equations  (57),  we 
choose  the  only  one  that  does  not  result  in  any  rotational  velocity  along  the 
fixation  point  vector  r0.  Mathematically  this  means  that  12  •  r0  =  0  which 
results  in  the  following  constraint  on  the  components  of  12 

x0 fix  T  y<>Qy  T  =  0.  (58) 

Given  the  fixation  velocity  (u0.  v0)  and  the  fixation  point  coordinates  x0  and 
y0,  the  equivalent  rotational  velocity  Q  is  obtained  by  solving  the  combination 
of  three  linear  equations  in  (57)  and  (58).  In  the  case  that  the  fixation  point 
is  at  the  principal  point.  x0  =  y0  =  0.  the  equivalent  rotational  velocity  is 


12  =  (t’o.  —  u0.  0). 


(59) 
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Let’s  define  the  rotational  fixation  velocity  as 

=  (n*0,fiv#,nI#)  =  -n.  (60) 

In  other  words  is  equal  to  but  in  opposite  direction  of  the  equivalent 
rotational  velocity  fi  given  by  equations  (57)  and  (58).  The  2nd  fixated 
image  can  be  obtained  by  applying  to  the  initial  2nd  image.  Considering 
eqn.  (57),  the  following  set  of  equations  must  be  satisfied  in  the  shifting 
process  of  the  initial  2nd  image 

f  u  =  xyVt Io  —  (x2  -f-  l)fiv#  ~h  yClZo 

\  w  =  (y2  +  l)Hx0  ~  xyfly.  -  xQx,. 

Here  f2Xo,  ilVo  and  f2Zo  are  known,  as  a  result  the  shifting  vector  (u,v)  can 
be  obtained  for  every  pixel  of  the  initial  2nd  image.  The  brightness  at  pixel 
(x,y)  of  the  fixated  2nd  image  is  obtained  by  finding  the  brightness  at  the 
corresponding  original  point  (x  —  Tu,y  —  Tv)  in  the  initial  2nd  irr^ge.  T  is 
the  time  interval  between  two  initial  images.  In  general  a  computed  original 
point  is  not  located  at  the  center  of  a  pixel  in  the  initial  2nd  image.  As  a 
result,  its  brightness  cannot  be  read  directly  from  the  image  file  and  should 
be  computed  by  averaging,  bilinear  interpolation  or  bicubic  interpolation  of 
the  brightnesses  at  its  neighboring  pixels. 

7  CONCLUSIONS  AND  FUTURE  WORK 

The  algorithms  and  formulations  presented  in  this  fixation  method  show 
how  to  solve  the  motion  vision  problem  directly  for  arbitrary  motion  with 
respect  to  an  arbitrary  rigid  scene.  In  contrast  to  previous  work  done  in  the 
area  of  motion  vision,  this  solution  is  general  and  does  not  put  any  severe 
restriction  on  the  motion  or  the  shape  of  environment.  More  importantly  the 
fixation  method  uses  neither  optical  flow  nor  feature  correspondence ;  instead 
direct  image  information  such  as  temporal  and  spatial  brightness  gradients 
are  used.  There  is  no  restriction  on  choosing  the  fixation  point.  However 
using  the  principal  point  as  the  fixation  point  makes  the  equations  more 
concise  and  the  calculations  easier. 

Implementation  of  this  fixation  method,  which  will  be  our  next  work,  is 
essential  for  supporting  the  feasibility  of  the  scheme.  Referring  to  fig.  1,  we 
can  implement  the  fixation  method  in  the  following  steps. 
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STEP  1:  Finding  the  fixation  velocity  components  (u0,v0)  and  component 
of  rotational  velocity  along  R0,  u>r0,  by  applying  the  system  of  eqn.  (51)  to 
the  direct  image  information  from  two  initial  images. 

STEP  2:  Knowing  the  fixation  velocity  components,  u0  and  u0,  the  fixated 
2nd  image  is  obtained  by  the  shifting  method  explained  in  section  6. 

STEP  3:  Using  the  general  fixation  constraint  equation  (25),  original  1st 
image,  and  the  fixated  2nd  image,  the  method  presented  in  section  4  can  be 
used  for  recovering  the  depth,  the  translational  velocity  and  the  rotational 
velocity.  Note  that  we  had  to  derive  the  special  fixation  constraint  equation 
(34)  separately,  but  for  implementation  it  is  enough  to  use  just  the  general 
fixation  constraint  equation  (25)  because  eqn.  (34)  is  a  special  case  of  eqn. 
(25).  As  a  result,  the  general  algorithms  can  be  used  for  recovering  the  motion 
and  depth,  without  knowing  in  advance  whether  the  present  condition  is  a 
special  case  or  not. 

STEP  f:  The  total  rotational  velocity  u)tot  is  simply  obtained  by  adding 
the  equivalent  rotational  velocity  fl,  from  equations  (57)  and  (58),  to  the 
rotational  velocity  u  from  eqn.  (46). 
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